Amazon Kinesis Data Streams FAQs

With Kinesis Data Streams, you can build custom applications that process or analyze streaming data for specialized needs. You can add various types of data such as clickstreams, application logs, and social media to a Kinesis data stream from hundreds of thousands of sources. Within seconds, the data will be available for your applications to read and process from the stream.

Kinesis Data Streams manages the infrastructure, storage, networking, and configuration needed to stream your data at the level of your data throughput. You don't have to worry about provisioning, deployment, or ongoing maintenance of hardware, software, or other services for your data streams. In addition, Kinesis Data Streams synchronously replicates data across three Availability Zones, providing high availability and data durability. By default, Kinesis Data Streams scales capacity automatically, freeing you from provisioning and managing capacity. You can choose provisioned mode if you want to provision and manage throughput on your own.

Kinesis Data Streams is useful for rapidly moving data off data producers and then continuously processing the data, whether that means transforming it before emitting to a data store, running real-time metrics and analytics, or deriving more complex data streams for further processing.

The following are typical scenarios for using Kinesis Data Streams:

• Accelerated log and data feed intake: Instead of waiting to batch the data, you can have your data producers push data to a Kinesis data stream as soon as the data is produced, preventing data loss in case of producer failure. For example, system and application logs can be continuously added to a data stream and be available for processing within seconds.
• Real-time metrics and reporting: You can extract metrics and generate reports from Kinesis data stream data in real time. For example, your Amazon Kinesis application can work on metrics and reporting for system and application logs as the data is streaming in, rather than waiting to receive data batches.
• Real-time data analytics: With Kinesis Data Streams, you can run real-time streaming data analytics. For example, you can add clickstreams to your Kinesis data stream and have your Kinesis application run analytics in real time, allowing you to gain insights from your data in minutes instead of hours or days.
• Log and event data collection: Collect log and event data from sources such as servers, desktops, and mobile devices. You can then build applications using Amazon Lambda or Amazon Managed Service for Apache Flink to continuously process the data, generate metrics, power live dashboards, and emit aggregated data into stores such as Amazon Simple Storage Service (Amazon S3).
• Power event-driven applications: Quickly pair with AWS Lambda to respond or adjust to immediate occurrences within the event-driven applications in your environment, at any scale.

After you sign up for AWS, you can start using Kinesis Data Streams by creating a Kinesis data stream through either the AWS Management Console or the CreateStream operation. Then configure your data producers to continuously add data to your data stream. You can optionally send data from existing resources in AWS services such as Amazon DynamoDB, Amazon Aurora, Amazon CloudWatch, and AWS IoT Core. You can then use AWS Lambda, Amazon Managed Service for Apache Flink, or AWS Glue Streaming to quickly process data stored in Kinesis Data Streams. You can also build custom applications that run on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), and Amazon Elastic Kubernetes Service (Amazon EKS) using either Amazon Kinesis API or Amazon Kinesis Client Library (KCL).

A shard has a sequence of data records in a stream. It serves as a base throughput unit of a Kinesis data stream. A shard supports 1 MB/second and 1,000 records per second for writes and 2 MB/second for reads. The shard limits ensure predictable performance, making it easy to design and operate a highly reliable data streaming workflow. A producer puts data records into shards and a consumer gets data records from shards. Consumers use shards for parallel data processing and for consuming data in the exact order in which they are stored. If writes and reads exceed the shard limits, the producer and consumer applications will receive throttles, which can be handled through retries.

A record is the unit of data stored in an Amazon Kinesis data stream. A record is composed of a sequence number, partition key, and data blob. Data blob is the data of interest your data producer adds to a data stream. The maximum size of a data blob (the data payload before Base64-encoding) is 1 megabyte (MB).

A partition key is used to isolate and route records to different shards of a data stream. A partition key is specified by your data producer while adding data to a Kinesis data stream. For example, let’s say you have a data stream with two shards (shard 1 and shard 2). You can configure your data producer to use two partition keys (key A and key B) so that all records with key A are added to shard 1 and all records with key B are added to shard 2.

A sequence number is a unique identifier for each record. Sequence number is assigned by Amazon Kinesis when a data producer calls PutRecord or PutRecords operation to add data to a Amazon Kinesis data stream. Sequence numbers for the same partition key generally increase over time; the longer the time period between PutRecord or PutRecords requests, the larger the sequence numbers become.

The capacity mode of Kinesis Data Streams determines how capacity is managed and usage is charged for a data stream. You can choose between provisioned and on-demand modes. In provisioned mode, you specify the number of shards for the data stream. The total capacity of a data stream is the sum of the capacities of its shards. You can increase or decrease the number of shards in a data stream as needed, and you pay for the number of shards at an hourly rate. In on-demand mode, AWS manages the shards to provide the necessary throughput. You pay only for the actual throughput used, and Kinesis Data Streams automatically accommodates your workload throughput needs as they ramp up or down. All Kinesis Data Streams write and read APIs, along with optional features such as Extended Retention and Enhanced Fan-Out, are supported in both capacity modes.

On-demand mode is best suited for workloads with unpredictable and highly variable traffic patterns. You should use this mode if you prefer AWS to manage capacity on your behalf or prefer pay-per-throughput pricing. Provisioned mode is best suited for predictable traffic, where capacity requirements are easy to forecast. You should consider using provisioned mode if you want fine-grained control over how data is distributed across shards. Provisioned mode is also suitable if you want to provision additional shards so the consuming application can have more read throughput to speed up the overall processing.

Yes. You can switch between on-demand and provisioned mode twice a day. The shard count of your data stream remains the same when you switch from provisioned mode to on-demand mode and vice versa. With the switch from provisioned to on-demand capacity mode, your data stream retains whatever shard count it had before the transition. But from that point on, Kinesis Data Streams monitors your data traffic and scales the shard count of this on-demand data stream up or down depending on traffic increase or decrease.

You can add data to a Kinesis data stream through PutRecord and PutRecords operations, KPL, or Amazon Kinesis Agent.

PutRecord operation allows a single data record within an API call, and PutRecords operation allows multiple data records within an API call. For more information, see PutRecord and PutRecords.

KPL is an easy-to-use and highly configurable library that helps you put data into an Amazon Kinesis data stream. KPL presents a simple, asynchronous, and reliable interface that helps you quickly achieve high producer throughput with minimal client resources.

Amazon Kinesis Agent is a prebuilt Java application that offers an easy way to collect and send data to your Amazon Kinesis data stream. You can install the agent on Linux-based server environments such as web servers, log servers, and database servers. The agent monitors certain files and continuously sends data to your data stream. For more information, see Writing with Agents.

Your data blob, partition key, and data stream name are required parameters of a PutRecord or PutRecords call. The size of your data blob (before Base64 encoding) and partition key will be counted against the data throughput of your Amazon Kinesis data stream, which is determined by the number of shards within the data stream.

A consumer is an application that processes all data from a Kinesis data stream. You can choose between shared fan-out and enhanced fan-out consumer types to read data from a Kinesis data stream. The shared fan-out consumers all share a shard’s 2 MB/second of read throughput and five transactions per second limits and require the use of the GetRecords API. An enhanced fan-out consumer gets its own 2 MB/second allotment of read throughput, allowing multiple consumers to read data from the same stream in parallel, without contending for read throughput with other consumers. You need to use the SubscribeToShard API with the enhanced fan-out consumers. We recommend using enhanced fan-out consumers if you want to add more than one consumer to your data stream.

You can use managed services such as AWS Lambda, Amazon Managed Service for Apache Flink, and AWS Glue to process data stored in Kinesis Data Streams. These managed services take care of provisioning and managing the underlying infrastructure so you can focus on writing your business logic. You can also deliver data stored in Kinesis Data Streams to Amazon S3, Amazon OpenSearch Service, Amazon Redshift, and custom HTTP endpoints using its prebuilt integration with Kinesis Data Firehose. You can also build custom applications using Amazon Kinesis Client Library, a prebuilt library, or the Amazon Kinesis Data Streams API.

KCL for Java, Python, Ruby, Node.js, and .NET is a prebuilt library that helps you easily build Amazon Kinesis applications for reading and processing data from an Amazon Kinesis data stream.

KCL handles complex issues such as adapting to changes in data stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault tolerance. KCL enables you to focus on business logic while building applications. Refer to Kinesis Data Streams documentation here for more details on KCL.

The SubscribeToShard API is a high-performance streaming API that pushes data from shards to consumers over a persistent connection without a request cycle from the client. The SubscribeToShard API uses the HTTP/2 protocol to deliver data to registered consumers whenever new data arrives on the shard, typically within 70 milliseconds, offering approximately 65% faster delivery compared to the GetRecords API. The consumers will enjoy fast delivery even when multiple registered consumers are reading from the same shard.

Enhanced fan-out is an optional feature for Kinesis Data Streams consumers that provides logical 2 MB/second throughput pipes between consumers and shards. This allows you to scale the number of consumers reading from a data stream in parallel, while maintaining high performance.

You should use enhanced fan-out if you have, or expect to have, multiple consumers retrieving data from a stream in parallel, or if you have at least one consumer that requires the use of the SubscribeToShard API to provide sub-200 millisecond data delivery speeds between producers and consumers.

Consumers use enhanced fan-out by retrieving data with the SubscribeToShard API. The name of the registered consumer is used within the SubscribeToShard API, which leads to utilization of the enhanced fan-out benefit provided to the registered consumer.

Yes. You can have multiple consumers using enhanced fan-out and others not using enhanced fan-out at the same time. The use of enhanced fan-out does not impact the limits of shards for traditional GetRecords usage.

Yes. To use SubscribeToShard, you need to register your consumers, which activates enhanced fan-out. By default, your consumer will use enhanced fan-out automatically when data is retrieved through SubscribeToShard.

A new data stream created in on-demand mode has a quota of 4 MB/second and 4,000 records per second for writes. By default, these streams automatically scale up to 200 MB/second and 200,000 records per second for writes.

A data stream in on-demand mode accommodates up to double its previous peak write throughput observed in the last 30 days. As your data stream’s write throughput hits a new peak, Kinesis Data Streams scales the stream’s capacity automatically. For example, if your data stream has a write throughput that varies between 10 MB/second and 40 MB/second, Kinesis Data Streams will ensure that you can easily burst to double the peak throughput of 80 MB/second. Subsequently, if the same data stream sustains a new peak throughput of 50 MB/second, Data Streams will ensure that there is enough capacity to ingest 100 MB/second of write throughput. However, you will see “ProvisionedThroughputExceeded” exceptions if your traffic grows more than double the previous peak within a 15-minute duration. You need to retry these throttled requests.

On-demand mode’s aggregate read capacity increases proportionally to write throughput to ensure that consuming applications always have adequate read throughput to process incoming data in real time. You get at least twice the write throughput to read data using the GetRecords API. We recommend using one consumer with the GetRecord API so it has enough room to catch up when the application needs to recover from downtime. To add more than one consuming application, you need to use enhanced fan-out, which supports adding up to 20 consumers to a data stream using the SubscribeToShard API, with each having dedicated throughput.

The throughput of a Kinesis data stream in provisioned mode is designed to scale without limits by increasing the number of shards within a data stream.

You can scale up a Kinesis data stream capacity in provisioned mode by splitting existing shards using the SplitShard API. You can scale down capacity by merging two shards using the MergeShard API. Alternatively, you can use UpdateShardCount API to scale up (or down) a stream capacity to a specific shard count.

The throughput of a Kinesis data stream is determined by the number of shards within the data stream. Follow the steps below to estimate the initial number of shards your data stream needs in provisioned mode. Note that you can dynamically adjust the number of shards within your data stream through resharding.

Estimate the average size of the record written to the data stream in kilobytes (KB), rounded up to the nearest 1 KB. (average_data_size_in_KB)

Estimate the number of records written to the data stream per second. (number_of_records_per_second)

Decide the number of Amazon Kinesis Applications consuming data concurrently and independently from the data stream. (number_of_consumers)

Calculate the incoming write bandwidth in KB (incoming_write_bandwidth_in_KB), which is equal to the average_data_size_in_KB multiplied by the number_of_records_per_second.

Calculate the outgoing read bandwidth in KB (outgoing_read_bandwidth_in_KB), which is equal to the incoming_write_bandwidth_in_KB multiplied by the number_of_consumers.

You can then calculate the initial number of shards (number_of_shards) your data stream needs using the following formula: number_of_shards = max (incoming_write_bandwidth_in_KB/1000, outgoing_read_bandwidth_in_KB/2000)

The throughput of a Kinesis data stream is designed to scale without limits. The default shard quota is 20,000 shards per stream for the following AWS Regions: US East (N. Virginia), US West (Oregon), and Europe (Ireland). For all other Regions, the default shard quota is 1,000 or 6,000 shards per stream. You can view your shard quota and request an increase via the AWS Service Quotas console.

In provisioned mode, the capacity limits of a Kinesis data stream are defined by the number of shards within the data stream. The limits can be exceeded either by data throughput or by the number of PUT records. While the capacity limits are exceeded, the put data call will be rejected with a ProvisionedThroughputExceeded exception. If this is due to a temporary rise of the data stream’s input data rate, retry by the data producer will eventually lead to completion of the requests. If it’s due to a sustained rise of the data stream’s input data rate, you should increase the number of shards within your data stream to provide enough capacity for the put data calls to consistently succeed. In both cases, Amazon CloudWatch metrics allow you to learn about the change of the data stream’s input data rate and the occurrence of ProvisionedThroughputExceeded exceptions.

In provisioned mode, the capacity limits of a Kinesis data stream are defined by the number of shards within the data stream. The limits can be exceeded either by data throughput or by the number of read data calls. While the capacity limits are exceeded, the read data call will be rejected with a ProvisionedThroughputExceeded exception. If this is due to a temporary rise of the data stream’s output data rate, retry by the Amazon Kinesis application will eventually lead to completion of the requests. If it’s due to a sustained rise of the data stream’s output data rate, you should increase the number of shards within your data stream to provide enough capacity for the read data calls to consistently succeed. In both cases, Amazon CloudWatch metrics allow you to learn about the change of the data stream’s output data rate and the occurrence of ProvisionedThroughputExceeded exceptions.

The default retention period of 24 hours covers scenarios where intermittent lags in processing require catch-up with the real-time data. A seven-day retention lets you reprocess data for up to seven days to resolve potential downstream data losses. Long-term data retention greater than seven days and up to 365 days lets you reprocess old data for use cases such as algorithm back testing, data store backfills, and auditing.

Yes. You can use the same getShardIterator, GetRecords, and SubscribeToShard APIs to read data retained for more than seven days. The consumers can move the iterator to the desired location in the stream, retrieve the shard map (including both open and closed), and read the records.

Yes. There are API enhancements to ListShards, GetRecords, and SubscribeToShard APIs. You can use the new filtering option with the TimeStamp parameter available in the ListShards API to efficiently retrieve the shard map and improve the performance of reading old data. The TimeStamp filter lets applications discover and enumerate shards from the point in time you wish to reprocess data and eliminate the need to start at the trim horizon. GetRecords and SubscribeToShards have a new field, ChildShards, which allows you to quickly discover the children shards when an application finishes reading data from a closed shard, instead of having to traverse the shard map again. The fast discovery of shards makes efficient use of the consuming application’s compute resources for any sized stream, irrespective of the data retention period.

You should consider the API enhancements if you plan to retain data longer and scale your stream’s capacity regularly. Stream scaling operations close existing shards and open new child shards. The data in all the open and closed shards is retained until the end of the retention period. So the total number of shards increase linearly with a longer retention period and multiple scaling operations. This increase in the shard map requires you to use ListShards with the TimeStamp filter and ChildShards field in GetRecords, and SubscribeToShard API for efficient discovery of shards for data retrieval. You will need to upgrade your KCL to the latest version (1.x for standard consumers and 2.x for enhanced fan-out consumers) for these features.

Yes. Clients of Kinesis Data Streams can use the AWS Glue Schema Registry, a serverless feature of AWS Glue, either through the KPL and KCL or through AWS Glue Schema Registry APIs in the AWS Java SDK. The Schema Registry is available at no additional charge. 

Visit the Schema Registry user documentation to get started and to learn more.

There are two ways to change the throughput of your data stream. You can use the UpdateShardCount API or the AWS Management Console to scale the number of shards in a data stream, or you can change the throughput of an Amazon Kinesis data stream by adjusting the number of shards within the data stream (resharding).

Typical scaling requests should take a few minutes to complete. Larger scaling requests will take longer than smaller ones.

Yes. You can continue adding data to and reading data from your Kinesis data stream while you use UpdateShardCount or reshard to change the throughput of the data stream or when Kinesis Data Streams does it automatically in on-demand mode.

The Amazon Kinesis Data Streams Management Console displays key operational and performance metrics such as throughput of data input and output of your Kinesis data streams. Kinesis Data Streams also integrates with Amazon CloudWatch so you can collect, view, and analyze CloudWatch metrics for your data streams and shards within those data streams. For more information about Kinesis Data Streams metrics, see Monitoring Amazon Kinesis Data Streams with Amazon CloudWatch.

Note that all stream-level metrics are free of charge. All enabled shard-level metrics are charged at Amazon CloudWatch pricing.

Kinesis Data Streams integrates with AWS Identity and Access Management (IAM), a service that helps you securely control access to your AWS services and resources for your users. For example, you can create a policy that allows only a specific user or group to add data to your data stream. You can also attach a resource-based policy to your data stream or registered consumer to control access at the resource level. For more information about access management and control of your data stream, see Controlling Access to Amazon Kinesis Data Streams Resources using IAM.

You can use IAM assume role or resource-based policy to share access with another account. To share access with a cross-account AWS Lambda function, attach a resource-based policy to your data stream or consumer to grant access to the the Lambda function's execution role. See more at more at Using AWS Lambda with Amazon Kinesis.

Kinesis Data Streams integrates with Amazon CloudTrail, a service that records AWS API calls for your account and delivers log files to you. For more information about API call logging and a list of supported Amazon Kinesis API operations, see Logging Amazon Kinesis API calls Using Amazon CloudTrail.

Kinesis Data Streams allows you to tag your Kinesis data streams and enhanced fan-out (EFO) consumers for easier resource and cost management. A tag is a user-defined label expressed as a key-value pair that helps organize AWS resources. For example, you can tag your Amazon Kinesis data streams or EFO consumers by cost centers so that you can categorize and track your Amazon Kinesis Data Streams costs based on cost centers. For more information about, see Tag your Amazon Kinesis Data Streams resources.

Amazon Kinesis is secure by default. Only the account and data stream owners have access to the Kinesis resources they create. Kinesis supports user authentication to control access to data. You can use IAM policies to selectively grant permissions to users and groups of users. You can securely put and get your data from Kinesis through SSL endpoints using the HTTPS protocol. If you need extra security, you can use server-side encryption with AWS Key Management Service (AWS KMS) keys to encrypt data stored in your data stream. AWS KMS allows you to use AWS-generated KMS keys for encryption, or if you prefer, you can bring your own KMS key into AWS KMS. Lastly, you can use your own encryption libraries to encrypt data on the client side before putting the data into Kinesis.

Yes. You can privately access Kinesis Data Streams APIs from your Amazon VPC by creating VPC Endpoints. With VPC Endpoints, the routing between the VPC and Kinesis Data Streams is handled by the AWS network without the need for an internet gateway, NAT gateway, or VPN connection. The latest generation of VPC Endpoints used by Kinesis Data Streams are powered by AWS PrivateLink, a technology that enables private connectivity between AWS services using Elastic Network Interfaces (ENI) with private IPs in your VPCs. To learn more about PrivateLink, visit the PrivateLink documentation.

Yes, and there are two options for doing so. You can use server-side encryption, which is a fully managed feature that automatically encrypts and decrypts data as you put and get it from a data stream. You can also write encrypted data to a data stream by encrypting and decrypting on the client side.

You might choose server-side encryption over client-side encryption for any of the following reason:

• It is hard to enforce client-side encryption.
• They want a second layer of security on top of client-side encryption.
• It is hard to implement client-side key management schemes.

Server-side encryption for Kinesis Data Streams automatically encrypts data using a user specified AWS KMS key before it is written to the data stream storage layer, and decrypts the data after it is retrieved from storage. Encryption makes writes impossible and the payload and the partition key unreadable unless the user writing or reading from the data stream has the permission to use the key selected for encryption on the data stream. As a result, server-side encryption can make it easier to meet internal security and compliance requirements governing your data.

With server-side encryption your client-side applications (producers and consumers) do not need to be aware of encryption, they do not need to manage KMS keys or cryptographic operations, and your data is encrypted when it is at rest and in motion within the Kinesis Data Streams service. All KMS keys used by the server-side encryption feature are provided by the AWS KMS. AWS KMS makes it easy to use an AWS-managed KMS key for Kinesis (a “one-click” encryption method), your own AWS KMS customer-managed key, or a KMS key that you imported for encryption.

Yes, there is a getting started guide in the user documentation.

Possibly. It depends on the key you use for encryption and the permissions governing access to the key.

• If you use the AWS-managed KMS key for Kinesis (key alias = aws/kinesis) your applications will not be impacted by enabling or disabling encryption with this key.
• If you use a different KMS key, like a custom AWS KMS key or one you imported into the AWS KMS service, and if your producers and consumers of a data stream do not have permission to use the KMS key used for encryption, then your PUT and GET requests will fail. Before you can use server-side encryption you must configure AWS KMS key policies to allow encryption and decryption of messages. For examples and more information about AWS KMS permissions, see AWS KMS API Permissions: Actions and Resources Reference in the AWS Key Management Service Developer Guide or the permissions guidelines in the Kinesis Data Streams server-side encryption user documentation.

Yes, however if you are using the AWS-managed KMS key for Kinesis and are not exceeding the AWS Free Tier KMS API usage costs, your use of server-side encryption is free. The following describes the costs by resource:

Keys:

The AWS-managed KMS key for Kinesis (alias = “aws/kinesis”) is free.
Customer managed KMS keys are subject to KMS key costs. Learn more.

KMS API Usage:

API usage costs apply for every KMS key, including custom ones. Kinesis Data Streams calls KMS approximately every five minutes when it’s rotating the data key. In a 30-day month, the total cost of KMS API calls initiated by a Kinesis data stream should be less than a few dollars. Note that this cost scales with the number of user credentials you use on your data producers and consumers because each user credential requires a unique API call to AWS KMS. When you use IAM role for authentication, each assume role-call will result in unique user credentials, and you might want to cache user credentials returned by the assume-role-call to save KMS costs.

Kinesis Data Streams server-side encryption is available in the AWS GovCloud Region and all public Regions except the China (Beijing) Region.

All of these operations can be completed using the AWS Management Console or the AWS SDK. To learn more, see the Kinesis Data Streams server-side encryption getting started guide.

Kinesis Data Streams uses an AES-GCM 256 algorithm for encryption.

No. Only new data written into the data stream will be encrypted (or left decrypted) by the new application of encryption.

Server-side encryption encrypts the payload of the message along with the partition key, which is specified by the data stream producer applications.

Server-side encryption is a stream specific feature.

Yes, using the AWS Management Console or the AWS SDK, you can choose a new KMS key to apply to a specific data stream.

No. Kinesis Data Streams is not currently available in the AWS Free Tier. AWS Free Tier is a program that offers free trial for a group of AWS services. For more details about AWS Free Tier, see AWS Free Tier.

Our Kinesis Data Streams SLA guarantees a Monthly Uptime Percentage of at least 99.9% for Kinesis Data Streams.

You are eligible for a SLA credit for Kinesis Data Streams under the Kinesis Data Streams SLA if more than one Availability Zone in which you are running a task, within the same Region has a Monthly Uptime Percentage of less than 99.9% during any monthly billing cycle.

For full details on all of the terms and conditions of the SLA, as well as details on how to submit a claim, please see Amazon Kinesis Data Streams SLA.

Kinesis Data Streams uses simple pay-as-you-go pricing. There are no upfront costs or minimum fees, and you pay only for the resources you use. Kinesis Data Streams has two capacity modes—on demand and provisioned—and both come with specific billing options.

With on-demand capacity mode, you don’t need to specify how much read and write throughput you expect your application to perform. In this mode, pricing is based on the volume of data ingested and retrieved along with a per-hour charge for each data stream in your account. There are additional charges for optional features: Extended data retention (beyond the first 24 hours and within the first seven days), Long-Term data retention (beyond seven days and up to one year), and Enhanced Fan-Out. For more information about Kinesis Data Streams costs, see Amazon Kinesis Data Streams Pricing.

With provisioned capacity mode, you specify the number of shards necessary for your application based on its write and read request rate. A shard is a unit of capacity that provides 1 MB/second of write and 2 MB/second of read throughout. You’re charged for each shard at an hourly rate. You also pay for records written into your Kinesis data stream. You incur additional charges when you use optional features such as Extended retention and Enhanced Fan-Out.

Following are two core dimensions and three optional dimensions in Kinesis Data Streams provisioned mode:

• Hourly Shard cost determined by the number of shards within your Amazon Kinesis data stream.
• PUT Payload Unit cost determined by the number of 25 KB payload units that your data producers add to your data stream.

Optional:

• Extended data retention is an optional cost determined by the number of shard hours incurred by your data stream. When extended data retention is enabled, you pay the extended retention rate for each shard in your stream.
• Long-term data retention is an optional cost with two cost dimensions: long-term data storage and long-term data retrieval. Long-term data storage reflects the numbers of GB-months data is stored for the period greater than seven days and up to 365 days. Long-term data retrieval reflects the number of GBs of data retrieved that has been stored for more than seven days.
• Enhanced fan-out is an optional cost with two cost dimensions: consumer-shard hours and data retrievals. Consumer-shard hours reflect the number of shards in a stream multiplied by the number of consumers using enhanced fan-out. Data retrievals are determined by the number of GBs delivered to consumers using enhanced fan-out.

For more information about Kinesis Data Streams costs, see Amazon Kinesis Data Streams Pricing.

A consumer-shard hour is calculated by multiplying the number of registered stream consumers with the number of shards in the stream. You will also pay only for the prorated portion of the hour the consumer was registered to use enhanced fan-out. For example, if a consumer-shard hour costs $0.015, for a 10-shard data stream, this consumer using enhanced fan-out would be able to read from 10 shards, and thus incur a consumer-shard hour charge of $0.15 per hour (1 consumer * 10 shards * $0.015 per consumers-shard hour). If there were two consumers registered for enhanced fan-out simultaneously, the total consumer-shard hour charge would be $0.30 per hour (2 consumers * 10 shards * $0.015).

Kinesis Data Streams and Amazon MSK are both popular data streaming platforms that help you build your own streaming workloads that process data for specialized needs. Both services are scalable, secure, and highly available. They can both be deployed to run streaming use cases such as real-time web and log analytics, personalizing customer experiences, event-driven architectures, IoT analytics, and real-time fraud detection. When choosing between the two, it is important to consider your specific use case and requirements. Here are some factors to consider:

Familiarity

•  If you are new to streaming technologies, use Kinesis Data Streams.
• If you have existing applications that are running on Apache Kafka, use MSK. MSK has an existing Kafka migration program (KMP) and a migration guide to make the migration experience easy.

Preference for open-source

• If you have a preference for using open-source technologies, our recommendation is to use MSK. Both MSK and MSK Connect are fully compatible with open-source Apache Kafka and Kafka Connect, respectively.

Kinesis Data Streams enables real-time processing of streaming big data. It provides ordering of records, as well as the ability to read and/or replay records in the same order to multiple Amazon Kinesis Applications. The Amazon Kinesis Client Library (KCL) delivers all records for a given partition key to the same record processor, making it easier to build multiple applications reading from the same Kinesis data stream (for example, to perform counting, aggregation, and filtering). Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable hosted queue for storing messages as they travel between computers. Amazon SQS lets you easily move data between distributed application components and helps you build applications in which messages are processed independently (with message-level ack/fail semantics), such as automated workflows.

We recommend Kinesis Data Streams for use cases with requirements that are similar to the following:

• Routing related records to the same record processor (as in streaming MapReduce). For example, counting and aggregation are simpler when all records for a given key are routed to the same record processor.
• Ordering of records. For example, you want to transfer log data from the application host to the processing/archival host while maintaining the order of log statements.
• Ability for multiple applications to consume the same stream concurrently. For example, you have one application that updates a real-time dashboard and another that archives data to Amazon Redshift. You want both applications to consume data from the same stream concurrently and independently.
• Ability to consume records in the same order a few hours later. For example, you have a billing application and an audit application that runs a few hours behind the billing application. Because Kinesis Data Streams stores data for up to 365 days, you can run the audit application up to 365 days behind the billing application.

We recommend Amazon SQS for use cases with requirements that are similar to the following:

• Messaging semantics (such as message-level ack/fail) and visibility timeout. For example, you have a queue of work items and want to track the successful completion of each item independently. Amazon SQS tracks the ack/fail so the application doesn’t have to maintain a persistent checkpoint/cursor. Amazon SQS will delete acked messages and redeliver failed messages after a configured visibility timeout.
• Individual message delay. For example, you have a job queue and need to schedule individual jobs with a delay. With Amazon SQS, you can configure individual messages to have a delay of up to 15 minutes.
• Dynamically increasing concurrency/throughput at read time. For example, you have a work queue and want to add more readers until the backlog is cleared. With Kinesis Data Streams, you can scale up to a sufficient number of shards (note, however, that you’ll need to provision enough shards ahead of time).
• Using the ability of Amazon SQS to scale transparently. For example, you buffer requests and the load changes as a result of occasional load spikes or the natural growth of your business. Because each buffered request can be processed independently, Amazon SQS can scale transparently to handle the load without any provisioning instructions from you.